#2597)

* refactor(delphi): delete dead scalar paths in repness.py (PR 14a)

The scalar implementations in `delphi/polismath/pca_kmeans_rep/repness.py`
were test-only — production calls only `compute_group_comment_stats_df`,
`select_rep_comments_df`, and `select_consensus_comments_df` (via
`conv_repness`). Maintaining two parallel implementations created
"where do I put this helper?" ambiguity for the upcoming D10/D11/D12 fixes
(all of which add new helpers to `repness.py`) and obscured the structure
of the production path.

Foundation pass before D10/D11/D12 + PR 14b/14c. No behavioural change —
production path untouched.

## Production code (`delphi/polismath/pca_kmeans_rep/repness.py`)

DELETED (445 lines):
- Primitives: `prop_test`, `two_prop_test` (no production callers).
- Orchestration: `comment_stats`, `add_comparative_stats`, `repness_metric`,
  `finalize_cmt_stats`, `passes_by_test`, `best_agree`, `best_disagree`,
  `select_rep_comments`, `select_consensus_comments`.
- Unused helper: `calculate_kl_divergence` (no callers anywhere in repo).

KEPT:
- `z_score_sig_90`, `z_score_sig_95` — trivial threshold checks consumed
  scalar-side; vectorizing would not save lines.

ENRICHED:
- `prop_test_vectorized` and `two_prop_test_vectorized` docstrings now embed
  the scalar-equivalent closed-form algebra. The formulas stay readable even
  though the scalar functions are gone.

## Tests

DELETED entirely:
- `tests/test_old_format_repness.py` (557 lines, scalar-only sibling of
  `test_repness_unit.py`).

DELETED classes/methods in `tests/test_repness_unit.py`:
- `TestCommentStats`, `TestSelectionFunctions`, `TestConsensusAndGroupRepness`
  (all scalar-only).
- `TestStatisticalFunctions::test_prop_test`, `::test_two_prop_test`.

MIGRATED to single vectorized DataFrame calls in `tests/test_discrepancy_fixes.py`:
- D4/D5/D6 BlobInjection classes — build a DataFrame from the blob's `repness`
  entries, run one `prop_test_vectorized` / `two_prop_test_vectorized` call,
  compare element-wise. Tests the actual production code path, produces
  cleaner diagnostics via `.to_string()`.
- `TestD5ProportionTest::test_prop_test_matches_clojure_formula` (consolidated
  with the n=0 boundary case).
- `TestD6TwoPropTest::test_two_prop_test_matches_clojure_formula` (+ edge cases
  consolidated, + pi_hat=1 boundary cases).

CONSOLIDATED in `tests/test_discrepancy_fixes.py`:
- `TestD8FinalizeStats`'s 7 scalar boundary tests collapse into a single
  parametrized DataFrame test `test_repful_classification_boundary` that
  exercises the production `np.where(rat > rdt, 'agree', 'disagree')` logic.
  All boundary cases preserved (rat<rdt, rat>rdt, rat==rdt non-zero,
  rat==rdt==0, negative z-scores).

DELETED redundant tests:
- `TestSyntheticEdgeCases::test_prop_test_matches_clojure_formula_synthetic`
  (duplicated by migrated TestD5ProportionTest).
- `TestSyntheticEdgeCases::test_clojure_repness_metric_product` (duplicated
  by migrated TestD7RepnessMetric).
- `TestSyntheticEdgeCases::test_clojure_repful_uses_rat_vs_rdt` (purely
  tautological).

Cross-checks in `tests/test_repness_unit.py::TestVectorizedFunctions` now
use `_prop_test_reference` and `_two_prop_test_reference` closed-form
staticmethods in place of scalar calls.

## Suite delta

- Pre (edge @ 2dce7385f): 330 passed, 12 skipped, 58 xfailed.
- Post:                    295 passed, 12 skipped, 58 xfailed.
- Delta: -35 passed, 0 failed, 0 new xfailed. Matches deleted scalar-test
  count exactly.

## For PR 14c (readability refactor — runs later)

The deleted scalar code is the readability reference for PR 14c. Retrieve via:

    git show <this-commit>~1:delphi/polismath/pca_kmeans_rep/repness.py \
        | sed -n '161,302p'

Specifically (pre-deletion line numbers): `comment_stats` 161-201,
`add_comparative_stats` 203-235, `repness_metric` 237-271,
`finalize_cmt_stats` 273-301. Clojure originals at
`math/src/polismath/math/repness.clj:78-100,173-188,191-200`.

## Documentation

- `delphi/docs/PLAN_DISCREPANCY_FIXES.md`: added PR 14a row to the stack
  cross-reference table.
- `delphi/docs/CLJ-PARITY-FIXES-JOURNAL.md`: appended "Session: PR 14a —
  Scalar deletion (2026-06-11)" entry with the full scope, suite delta,
  and the `git show` recipe for PR 14c.

## Out of scope (handed off separately)

- Pyright pandas-stubs noise (10+ false positives on `pd.DataFrame(columns=...)`
  and `df['col'] = value` in `compute_group_comment_stats_df`) is pre-existing
  on edge HEAD; PR #2560's pyright config didn't set rule overrides.
  Handoff: `~/polis/HANDOFF_PYRIGHT_PANDAS_STUBS.md`. Tracked separately.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

commit-id:694a6768

* fix(delphi): ns includes PASS votes (Clojure parity, pre-D10)

Bug
---
`compute_group_comment_stats_df` in
`delphi/polismath/pca_kmeans_rep/repness.py` computed `ns = na + nd` and
`total_votes = total_agree + total_disagree`, silently dropping PASS (0)
votes from every vote count.

Clojure reference: `math/src/polismath/math/repness.clj:56-61, :70`:

    (defn- count-votes [votes & [vote]]
      (let [filt-fn (if vote #(= vote %) identity)]
        (count (filter filt-fn votes))))
    ...
    :ns (fnk [votes] (count-votes votes))

`count-votes` with no `vote` argument uses `identity` as the filter
predicate. In Clojure 0 is truthy, so `(filter identity ...)` keeps every
non-nil entry — including PASS. Therefore Clojure
`ns = na + nd + np` (PASS count).

Every downstream metric that consumes `ns` — `pa`, `pd`, `pat`, `pdt`,
`ra`, `rd`, `rat`, `rdt`, `agree_metric`, `disagree_metric`, and the
upcoming D11 `consensus_stats_df` — was off whenever PASS votes existed.

Fix
---
Both `total_counts` and `group_counts` now compute the count column via
`('vote', 'size')` directly in the pandas groupby aggregation. The frame
is `dropna(subset=['vote'])`-filtered upstream, so `size()` counts
exactly the non-NaN rows — including PASS (0). This is the Clojure
`(count (filter identity votes))` recipe verbatim. Both sites carry a
comment citing the Clojure reference.

Why D5 BlobInjection didn't catch this
--------------------------------------
D5's blob-injection tests pull `(n-success, n-trials)` directly from the
Clojure blob's `repness` entries and feed them to `prop_test_vectorized`.
They bypass `compute_group_comment_stats_df` entirely, so anything
downstream of `ns = na + nd` was invisible. The same gap will recur for
every fix whose stage-inputs are built by Python code. Pure-formula
tests over a tiny vote matrix are the only RED path.

Tests added
-----------
`TestNsIncludesPassVotes` in `tests/test_repness_unit.py`:

- `test_ns_includes_pass_votes` — single comment, 2 agree / 1 disagree /
  2 pass → `na=2, nd=1, ns=5`.
- `test_ns_all_pass_column` — all-PASS column → `na=0, nd=0, ns=3`.
- `test_ns_mixed_with_nan_only_explicit_votes_count` — NaN never counts;
  PASS always does.
- `test_other_votes_includes_other_group_pass` — `other_votes` (the
  complement of group `ns`) also includes out-of-group PASS.

Suite delta
-----------
Pre: 295 passed, 12 skipped, 58 xfailed (PR 14a baseline).
Post: 299 passed, 12 skipped, 58 xfailed. Delta = +4 (the new tests).
No pre-existing test broke — the existing `TestVectorizedFunctions`
fixtures used only AGREE/DISAGREE/NaN and were never sensitive.

Cascade to D11
--------------
D11's `consensus_stats_df(vote_matrix_df)` (whole-conversation
counterpart) will mirror the same recipe at the conversation level. This
fix lands BEFORE D10 in the stack so D11's implementation, when it
arrives, starts from the corrected ns semantics. D11 should follow the
same pure-formula test pattern.

Goldens
-------
DEFERRED. Re-recording is gated on sklearn-KMeans-seeding consensus; no
golden values shift at the goldens commit until D10/D11/D12 land.

commit-id:0761e6c3

* feat(delphi): D10 — Clojure-parity rep comment selection (PR 8)

Replaces the pre-D10 botched-port `select_rep_comments_df` with a
single-pass reduce that mirrors Clojure `select-rep-comments`
(math/src/polismath/math/repness.clj:212-281).

Sits on top of PR 14a in the spr stack.

## Helpers added (top-level in `repness.py`)

- `passes_by_test(s)` — Clojure `passes-by-test?` (repness.clj:165-170).
  OR'd on (rat, pat) and (rdt, pdt) z-sig-90. NO `pa >= 0.5` gate; the
  pre-D10 Python gate was an over-restriction with no Clojure analog.

- `beats_best_by_test(s, current_best_z)` — Clojure `beats-best-by-test?`
  (repness.clj:133-139). Strict `>` on `max(rat, rdt)`.

- `beats_best_agr(s, current_best)` — Clojure `beats-best-agr?`
  (repness.clj:142-162). Four-branch agree-priority logic:
    1. na == 0 AND nd == 0 → reject.
    2. current_best AND current_best.ra > 1.0 → compare 4-way signed product
       `ra * rat * pa * pat`.
    3. current_best (else, ra <= 1.0) → compare `pa * pat` only.
    4. No current_best → accept if `z90(pat)` OR `(ra > 1.0 AND pa > 0.5)`.
  `current_best` stores the RAW row (Clojure repness.clj:250) so the
  ra/rat/pa/pat surface stays available across iterations.

- `_finalize_row_for_output(row, *, is_best_agree=False)` — Clojure
  `finalize-cmt-stats` (repness.clj:173-188) + best-agree flagging
  (repness.clj:262-264). Emits `best_agree=True` and `n_agree=na` for the
  best-agree slot.

## `select_rep_comments_df` rewrite

Signature now: `(stats_df, mod_out=None) -> List[Dict[str, Any]]`. Drops
the `agree_count` / `disagree_count` kwargs (Clojure has only a cap of 5).

Per-row state `{sufficient, best, best_agree}` updated by the helpers.
Final assembly: dedup best_agree from sufficient → sort by metric
(agree_metric for repful=='agree', disagree_metric for 'disagree')
→ prepend finalized+flagged best_agree → take 5 → agrees-before-disagrees.

The caller in `conv_repness` drops the `_stats_row_to_dict` wrapping step
(the new function returns finalized dicts directly).

## Two pre-D10 bugs fixed alongside the rewrite

- `pa >= 0.5 / pd >= 0.5` over-gate in the passing filter — removed (no
  Clojure analog).
- "Fill from other category" + "first row" fallback blocks — deleted. The
  `:best` / `:best_agree` mechanism IS the Clojure fallback.

## Tests (18 new in `tests/test_discrepancy_fixes.py`)

- TestD10PassesByTest (4): agree-side, disagree-side, neither, no pa-gate.
- TestD10BeatsBestByTest (3): None-best, max(rat,rdt), strict `>`.
- TestD10BeatsBestAgr (6): one per Clojure branch + boundary.
- TestD10SelectRepCommentsBoundary (5): empty input, single unvoted row
  → best fallback, sufficient-empty-best-agree-only, take-5 cap +
  agrees-before-disagrees, **the eviction edge case** (best_agree outside
  sufficient evicting 5th-highest-metric).

## Eviction edge case — flagged

`take(5)` runs AFTER prepending best_agree. If `best_agree` was kept by
`beats_best_agr` as a non-significant agree-priority fallback (failed
`passes_by_test`, qualified via Branch 4) AND `:sufficient` already has 5
entries, the prepend pushes total to 6 and `take(5)` evicts the
5th-highest-metric sufficient entry — possibly a strong dissenting view.

Mirrors Clojure exactly for blob parity. `# TODO(parity-eviction)` comment
at the take(5) site in the production code; entry added under
"Pending — needs team discussion" in PLAN.md; pinned by the synthetic
test above.

## Re-xfailed with updated reasons (D14 / D1 upstream divergence)

Six per-shared-(gid, tid) blob-comparison tests were previously xfailed as
"D5/D6/D7/D8/D10: no shared comments to compare". After D10 there ARE
shared comments (overlap ~20% on vw cold_start), but the per-(gid, tid)
stats still mismatch because Python and Clojure place different
participants in the "same" group ID. That's upstream PCA/KMeans
group-membership divergence (D14 / D1), not D10. Updated xfail reasons
point at D14 / D1. D10 itself is verified by the 18 synthetic tests.

Affected: TestD9ZScoreThresholds::test_z_values_match_clojure,
TestD5ProportionTest::test_pat_values_match_clojure_blob,
TestD6TwoPropTest::test_rat_values_match_clojure_blob,
TestD7RepnessMetric::test_repness_metric_matches_clojure_blob,
TestD8FinalizeStats::test_repful_matches_clojure_blob,
TestD10RepCommentSelection::test_rep_comments_match_clojure.

## Suite delta

- Pre (post-14a baseline): 295 passed, 12 skipped, 58 xfailed.
- Post:                    313 passed, 12 skipped, 58 xfailed.
- Delta: +18 passed (the 18 new D10 synthetic tests), 0 failed, no new
  xfailed.

## Documentation

- `delphi/docs/PLAN_DISCREPANCY_FIXES.md`: PR 14a row marked landed
  (#2564), PR 8 (D10) marked in-flight, eviction concern added to
  "Pending — needs team discussion".
- `delphi/docs/CLJ-PARITY-FIXES-JOURNAL.md`: "Session: PR 8 — D10 rep
  comment selection (2026-06-11)" entry with full scope, suite delta,
  and decisions log pointer.

## /goal mode

This PR is part of an autonomous run (`/goal`) targeting D10 + D11 + D12 +
golden snapshots as a stacked PR series. Decisions made autonomously
(key naming, return type, etc.) are documented in
`~/polis/D10_D11_D12_GOLDENS_DECISIONS.md` for batch user review at the
end of the run.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

commit-id:1b681fef

* feat(delphi): D11 — Clojure-parity consensus comment selection (PR 9)

Replaces the pre-D11 consensus logic (per-group `pa > 0.6 for all` filter,
top 2 by `avg_pa`) with a whole-conversation per-comment-stats stage and
two independent top-5 lists (agree / disagree), matching Clojure
`consensus-stats` + `select-consensus-comments`
(math/src/polismath/math/repness.clj:284-323).

Sits on top of PR 8 (D10) in the spr stack.

## Production changes (`repness.py`)

- **`consensus_stats_df(vote_matrix_df, mod_out=None) -> pd.DataFrame`**:
  new helper. Whole-conversation per-comment stats (no group split). Output
  DataFrame indexed by tid with cols [na, nd, ns, pa, pd, pat, pdt].
  Vectorized port of Clojure `consensus-stats` (repness.clj:284-290).

  **`ns` includes PASS** (Clojure parity, repness.clj:56-61): computed as
  `vote_matrix_df.notna().sum(axis=0)` rather than `na + nd`. Matches
  Clojure `(count (filter identity ...))` where `0` (PASS) is truthy.

- **`select_consensus_comments_df` rewrite**: new signature
  `(cons_stats) -> Dict[str, List[Dict]]`. Filters and ordering:
    - agree: `pa > 0.5 AND z-sig-90(pat)`, sorted desc by `am = pa * pat`.
    - disagree: `pd > 0.5 AND z-sig-90(pdt)`, sorted desc by `dm = pd * pdt`.
  Cap: top 5 each side. Output: `{'agree': [...], 'disagree': [...]}`.
  With PSEUDO_COUNT=2, `pa + pd = 1` exact → `pa > 0.5 ⟺ pd < 0.5`, so
  the same tid cannot appear in both lists.

- **`conv_repness` grows `mod_out` kwarg**, forwarded to both
  `select_rep_comments_df` and `consensus_stats_df`. Consensus is now run
  unconditionally — Clojure has no `len(groups) > 1` guard.

- **`_stats_row_to_dict` deleted** — orphan after D11.

## Caller (`conversation.py`)

`_compute_repness` passes `mod_out=self.mod_out_tids` to `conv_repness`,
matching Clojure's mod-out propagation (repness.clj:222 and :296).

## Downstream output shape change

`conv.repness['consensus_comments']` was a flat list with
`{repful: 'consensus', comment_id, avg_agree, stats}` entries. After D11
it's `{'agree': [entries], 'disagree': [entries]}` matching Clojure's
math-blob shape. Each entry has Python-convention keys (decision S1):
`{comment_id, n_success, n_trials, p_success, p_test}`.

Updated consumers in the test suite:
- `tests/test_repness_smoke.py::test_repness_structure` — iterates the
  new dict shape.
- `tests/test_pipeline_integrity.py::test_full_pipeline` — same.

External downstream consumers (`client-report/normalizeConsensus.js`) may
need a parallel update; flagged in the decisions log for batch review.

## Tests (13 new in `tests/test_discrepancy_fixes.py`)

- `TestD11ConsensusStatsDf` (5): basic counts, pseudocount pa/pd
  smoothing, ns=0 uninformative fallback, mod_out tid filter,
  **ns-includes-PASS Clojure parity**.
- `TestD11SelectConsensusBoundary` (8): empty input, clear agree
  consensus, clear disagree consensus, divisive (no consensus), top-5
  cap, entry-key shape, disagree-side key mapping (n_success ← nd,
  p_success ← pd, p_test ← pdt), mutually-exclusive agree/disagree lists.

## `ns`-PASS fix (Clojure parity)

After D11 was first landed (PR #2567), the real-data test
`test_consensus_matches_clojure` showed 3-5/5 overlap on cold_start.
Investigation revealed that Clojure's `:ns` (via `count-votes` with
`filter identity` — repness.clj:56-61) INCLUDES PASS votes (`0` is truthy
in Clojure), while Python's `ns = na + nd` excluded them.

Fixed here by switching `consensus_stats_df` to count via
`vote_matrix_df.notna().sum(axis=0)`. After the fix, 3 of 4 dataset
variants (vw-incremental, vw-cold_start, biodiversity-cold_start) match
Clojure exactly. The `biodiversity-incremental` variant still mismatches
on the disagree side (likely residual upstream PCA/KMeans group-membership
divergence) — remains `xfail(strict=False)` and tracked in the journal.

(The companion fix for `compute_group_comment_stats_df` ships in a
separate pre-D10 commit `qyskkqkovtmn`.)

## B1 + B2 sub-agent fixes (relocated from D12 per batch review 2026-06-11)

These two fixes were originally landed in PR #2568 (D12) because the D11
sub-agent review happened AFTER D11 had been pushed. They belong in D11,
so they are squashed into this commit:

- **B1** (`polismath/conversation/conversation.py:830-837`): the
  no-groups branch of `_compute_repness` returned
  `'consensus_comments': []` (list). After D11, the public shape is the
  dict `{'agree': [...], 'disagree': [...]}`. Downstream consumers
  (test_repness_smoke, test_pipeline_integrity) iterate the dict shape
  and would crash on the legacy list. Fixed to always return the dict
  shape, even with no groups.

- **B2** (`tests/test_legacy_repness_comparison.py:197-205`): the
  legacy-comparison test extracted `py_consensus = py_results.get(
  'consensus_comments', [])` and treated it as a flat list. Post-D11
  this is a dict, so the ID extraction silently produced an empty set.
  Fixed to flatten the dict (agree + disagree) for the legacy comparison,
  with a `legacy fallback` branch in case the value is still a list.

## Suite delta

- Pre (post-D10):    313 passed, 12 skipped, 58 xfailed.
- Post (this PR):    330 passed, 12 skipped, 55 xfailed, 3 xpassed.
- Delta: +17 passed, -3 xfailed (D11 real-data test now passes on 3 of
  4 dataset variants; biodiversity-incremental remains
  xfail(strict=False)). Zero regressions.

## /goal mode

Part of an autonomous stacked PR series (D10 + D11 + D12 + goldens) per
user request. Decisions are documented for batch review in
`~/polis/D10_D11_D12_GOLDENS_DECISIONS.md`.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

commit-id:bf7eabfe

* feat(delphi): D12 — comment priorities (PR 11)

Implements `comment-priorities` matching Clojure
(math/src/polismath/math/conversation.clj:648-679). Pre-D12 Python emitted
nothing for `comment_priorities`, which forced the TypeScript server's
`getNextPrioritizedComment` to fall back to uniform random comment routing.

Sits on top of PR 9 (D11) in the spr stack.

## New helpers in `pca.py`

- `pca_project_cmnts(center, comps) -> np.ndarray` (shape (n_cmnts, n_components)):
  Vectorized projection. Closed-form derived from Clojure's sparsity-aware
  projection (pca.clj:134-178) collapsing to a single non-nil column per
  comment:
      proj[i] = -sqrt(n_cmnts) * (1 + center[i]) * [pc1[i], pc2[i]]

- `compute_comment_extremity(cmnt_proj) -> np.ndarray`: L2 norm per row.
  Clojure `with-proj-and-extremtiy` (conversation.clj:341-352).

## New module-level functions in `conversation.py`

- `META_PRIORITY = 7` (Clojure conversation.clj:319).
- `importance_metric(A, P, S, E) -> float` (Clojure conversation.clj:311-315).
- `priority_metric(is_meta, A, P, S, E) -> float` (Clojure conversation.clj:321-330).
  Squared output. Meta: `META_PRIORITY^2 = 49`. Non-meta:
  `(importance * (1 + 8*2^(-S/5)))^2` — the decay factor lets new comments
  bubble up; importance falls as votes accumulate.

## New `Conversation._compute_comment_priorities()` method

Wired into `recompute()` after `_compute_repness()`. For each tid:
- Compute extremity from `pca_project_cmnts` + `compute_comment_extremity`.
- Aggregate A/D/S across all groups via `_compute_group_votes()`.
- Derive P = S - (A + D)  (Clojure conversation.clj:661).
- Check `tid in self.meta_tids` for the meta branch.
- Call `priority_metric` and store under `self.comment_priorities[int(tid)]`.

The serialization infrastructure (`to_dict`, `to_dynamo_dict`,
underscore→hyphen conversion in `_convert_inner`) already existed but was
emitting empty. Now populated.

## B1 + B2 fixes folded in from D11 sub-agent review

- **B1**: `conversation.py:834` no-groups early-return now emits
  `consensus_comments: {'agree': [], 'disagree': []}` (dict) instead of
  `[]` (list) — restores shape consistency with the new D11 shape.
- **B2**: `test_legacy_repness_comparison.py:197` flattens the new
  consensus dict before iterating, instead of crashing on
  `'agree'.get('comment_id', '')`.

## Tests (11 new in `tests/test_discrepancy_fixes.py`)

- `TestD12PCAProjectComments` (5): output shape, formula verification per
  row, empty inputs, L2 extremity, empty extremity.
- `TestD12PriorityMetrics` (6): `importance_metric` matches Clojure
  reference value `4/9` (from conversation.clj:335 comment), extremity
  boost behavior, meta priority constant = 49, non-meta squared formula,
  decay factor monotonicity (low-S boosts, high-S fades), `META_PRIORITY == 7`.

## Real-data test xfailed: Clojure blob has constant priorities

vw and biodiversity blobs both have EVERY tid set to priority = 49.0
(= META_PRIORITY^2). Likely caused by Clojure's `(if 0 ...)` truthiness
quirk — 0 is truthy in Clojure (only nil/false are falsy), so any value
returned by `(get meta-tids tid 0)` triggers the meta branch.

Python correctly distinguishes meta from non-meta via Boolean set
membership, producing varied priorities 0.18-31.46. Spearman comparison
meaningless when Clojure side has zero variance (returns nan).

Test xfailed with full reason. D12 logic verified by the 11 synthetic
tests. Logged for batch review — Python may be MORE correct than Clojure
here.

## Suite delta

- Pre (post-D11): 325 passed, 12 skipped, 58 xfailed.
- Post (this PR): 336 passed, 12 skipped, 56 xfailed, 2 xpassed.
- Delta: +11 (the 11 new D12 synthetic tests), 0 failed, 2 xfailed →
  xpassed (the cold_start D12 tests now run cleanly; the new xfail is
  on a different test).

## /goal mode

Autonomous stack PR (D10 + D11 + D12 + goldens). Decisions documented in
`~/polis/D10_D11_D12_GOLDENS_DECISIONS.md` for batch user review.

D11 sub-agent flagged additional concerns (B3: math-blob plumbing
hardcodes empty `consensus` in to_dict/to_dynamo_dict; D11 data
computed-but-unused at serialization layer). NOT addressed here —
larger scope than D11/D12, deliberate per S1. Documented for batch
review.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

commit-id:d02440bf

* feat(delphi): plumb D11 consensus + D12 priorities through to_dict / to_dynamo_dict

commit-id:ff4764e4

* fix(delphi): address Copilot review on D10-D12 stack (consensus blob keys, priority serialization, defensive fixes)

Verified triage of all 83 Copilot review threads on PRs #2564-#2573:
1 real blocker + 4 escalations confirmed by direct code verification;
the rest nitpicks / already-fixed / intentional bug-mirrors. All fixes
TDD RED->GREEN.

- Consensus entry keys -> Clojure blob shape {tid, n-success, n-trials,
  p-success, p-test} (was Python-convention comment_id/n_success/...).
  Narrows the S1 deferral: these entries flow raw into
  result['consensus'], where server-helpers.ts:298-313 and client-report
  majorityStrict.jsx:23-27 pluck `tid` - the Python keys broke both
  consumers. Rep-comment entries keep comment_id until the deferred
  math-blob alignment PR.
- DynamoDB writer read D11 consensus from a key to_dynamo_dict never
  emits (repness.consensus_comments) -> always wrote the empty default;
  D11 data never reached Delphi_PCAResults. Now reads top-level
  result['consensus']. The round-trip test had stubbed to_dynamo_dict
  with the writer's wrong nested shape - stub corrected to the real
  producer shape.
- to_dynamo_dict comment priorities: int(value) -> Decimal-preserving.
  int() floors sub-1 priorities to 0 (real formula spans ~0.18-31.46 per
  D12.6), which the TS server's weighted routing reads as "no priority
  data". Harmless today (bug-mirror pins 49.0), landmine once #2571
  resolves. Legacy writer path Decimal-wrapped too.
- DynamoDB reader normalizes legacy list-shaped consensus to the dict
  shape (pre-D11 blobs).
- mod_out truthiness -> `is not None` x2 in repness.py (numpy
  array/Index-safe).
- _compute_comment_priorities fails closed (error log + empty dict) on
  PCA/columns desync instead of silently zip-truncating extremities.
- bench_repness.py imported the 14a-deleted comment_stats (ImportError
  on any benchmark run); benchmark import test added.
- Per-variant xfails replace blanket xfail(strict=False) on D11/D12
  real-data tests, so the variants that match Clojure gate again.
  DISCOVERY: scoping the blanket unmasked two previously-invisible
  incremental divergences (bg2018-incremental, pakistan-incremental
  consensus vs Clojure) that the blanket had silently absorbed -
  documented as known-bad incremental xfails, same family as
  biodiversity-incremental, deferred to the sequential-parity work.
  3 pre-existing CCR failures (verified identical on edge 722640eb0)
  marked with precise per-variant reasons. PGR regression tests skip
  with the 2026-06-11 goldens-deferral reason - the mark S3-5 claimed
  to add but never committed.
- test_repness_smoke consensus-entry structure assertion updated to the
  Clojure key shape (exact key-set check).
- ns docstrings corrected (ns includes PASS post ns-PASS fix; pa+pd <= 1
  reasoning in select_consensus_comments_df).
- PERF deferral comment on the _compute_group_votes scan in
  _compute_comment_priorities (follow-up issue to be filed).

Baseline (2026-07-04, stack top, --include-local): 13 failed / 476
passed / 18 skipped / 143 xfailed. All 13 accounted for: 10 stale-PGR
comparisons (now skipped per deferral), 3 pre-existing CCR (now precise
xfails).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013CiESAwcrZ6BcVfdkumnxa

commit-id:4cc93a23

* fix(delphi): Decimal-convert D11 consensus for DynamoDB write

Caught by CI's test_math_pipeline_runs_e2e on the stack branch
(2026-07-05): "Float types are not supported. Use Decimal types
instead." at the Delphi_PCAResults put_item.

Once the writer read top-level `consensus` (the key to_dynamo_dict
actually emits — fixed in the PR below this one), REAL D11 consensus
data flowed to DynamoDB for the first time, carrying float
p-success/p-test values. Only the LEGACY writer branch Decimal-converted
its payload; the pre-formatted branch wrote `dynamo_data['consensus']`
raw into the Item, and boto3's TypeSerializer rejects raw floats.

Invisible to every local gate, three ways: the e2e test needs DynamoDB
(local skip list), the consensus round-trip tests use MagicMock (no
TypeSerializer runs), and the float-serialization pins covered
priorities but not consensus.

Fix, TDD RED->GREEN with boto3's REAL TypeSerializer:
- to_dynamo_dict: consensus surface now passes through float_to_decimal
  (same treatment as pca and comment_priorities in the same function).
- DynamoDB writer Site 1: belt-and-braces _replace_floats_with_decimals
  at the boto3 boundary, mirroring the legacy branch (idempotent on
  already-converted data).
- New TestToDynamoDictConsensusSerialization (Layer 2c) pins the exact
  failure through the real serializer; two mock-era expectations updated
  from float-identity to value-preserving comparisons.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013CiESAwcrZ6BcVfdkumnxa

commit-id:22c91871

* test(delphi): require_dynamodb skips locally, fails loudly in CI

DynamoDB-gated tests (test_math_pipeline_runs_e2e, test_batch_id) needed
a blanket --ignore in local runs because require_dynamodb hard-failed
when the service was absent. Now: pytest.skip locally with a how-to-run
hint (`docker run --rm -d -p 8002:8000 amazon/dynamodb-local` +
DYNAMODB_ENDPOINT), pytest.fail in CI — where DynamoDB is provisioned
and its absence is an infrastructure failure that must not silently
disable the only end-to-end gate (the 2026-07-05 consensus-float crash
was caught precisely because CI runs it).

Discriminator is GITHUB_ACTIONS, deliberately NOT the generic CI:
local supply-chain wrappers (SafeDep pmg) inject CI=true into wrapped
package-manager invocations, which would force the loud-fail path on
developer machines (observed 2026-07-05 on `uv run` via the pmg alias).

Verified matrix: local+down 5 skipped / local+up(8002) e2e PASSES
against real DynamoDB (the consensus-Decimal fix validated end-to-end)
/ GITHUB_ACTIONS+down errors loudly.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013CiESAwcrZ6BcVfdkumnxa

commit-id:bc3ca29e

* docs(delphi): archive 33 stale leftover docs, fix stale references

Audit of delphi/docs/ against the current code (2026-06-11). Per review
feedback (Colin): the stale docs are MOVED to delphi/docs/archive/ rather
than deleted — they capture the original research and design intent of the
2025 build-out, which is an independent deliverable worth keeping as raw
material for future models to mine. archive/CLAUDE.md marks the folder as
historical, instructs agents not to treat it as current documentation, and
indexes each file's original purpose and the reason it was archived.

- Move 33 stale docs to docs/archive/ byte-identical (git records renames):
  completed one-off fix memos and session logs, unimplemented design
  proposals (IGAS topic-consensus metrics, job-id migration, DAG job
  system, spatial topic prioritization, UMAP viz plan, 16-week
  topic-agenda migration), and docs describing deleted architecture
  (legacy poller, run_tests.py, simplified tests, eda_notebooks, 600/802
  scripts, custom power-iteration PCA).
- Amend 15 surviving docs: status headers on handoffs and
  deep-analysis-for-julien/07 (which D-fixes are merged vs open),
  deleted-script references (start_poller.sh, 600_*.py), table name
  DelphiJobQueue -> Delphi_JobQueue, uv instead of pip/venv,
  TopicAgenda.jsx -> .tsx, versioned-key delimiter correction.
- Rewrite DOCUMENTATION_DIRECTORY.md to index surviving docs + archive.
- Fix references to moved docs and deleted scripts in delphi/CLAUDE.md and
  DELPHI_JOB_SYSTEM_TROUBLESHOOTING.md (run_delphi.sh -> run_delphi.py,
  start_poller.sh -> start_poller.py, nonexistent reset_database.sh,
  dead absolute-path link to DATABASE_NAMING_PROPOSAL.md).

Kept in place deliberately, as they document still-unfixed bugs:
ZID_EXPOSURE_AUDIT.md (zid still exposed in delphi API responses) and
TOPIC_LABEL_MISALIGNMENT_ANALYSIS.md (700_datamapplot label sorting).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

commit-id:4336711e

* fix(delphi): keep Clojure group-id order — resolves the gid 0-1 label swap

Root cause of the label swap confirmed by the S3-4 trace (2026-06-11:
Python g0 = Clojure g1 with 50/50 identical membership on vw-cold_start):
_compute_clusters re-sorted group clusters by size (descending) and
reassigned ids. Clojure assigns group ids by first-k-distinct encounter
order over base-cluster centers (init-clusters, clusters.clj:55-64),
keeps them through merge lineage, and orders output by sort-by :id
(conversation.clj:437) - it never re-orders by size. The base-cluster
level already preserved k-means id order for exactly this reason (K-inv);
the group level now follows the same rule.

Fix: remove the size re-sort + id reassignment; keep k-means label order
(sklearn preserves init-index-to-label correspondence, and the base-center
row order is already Clojure-parity per K-inv). Pinned by a synthetic
test: with the smaller group's center encountered first, group id 0 must
be the smaller group (fails under any size re-sort).

Test-gate harvest, verified against a full --include-local run:
- TestD8FinalizeStats::test_repful_matches_clojure_blob: xfail LIFTED on
  9/11 variants (all but vw-incremental / pakistan-incremental, which
  keep per-variant xfails for residual incremental trajectory divergence).
- D9 significance-sets + D10 rep-selection: biodiversity-cold_start now
  matches Clojure exactly and gates; other variants keep per-variant
  xfails (residual per-(gid,tid) membership/stat divergence).
- z-values / rat-values xfail reasons corrected: the label swap is now
  FALSIFIED as their cause (fix did not flip them); residual cause is
  group-membership divergence.
- D12 priorities known-bad incremental lists refined: FLI-incremental and
  bg2050-incremental Clojure blobs carry the all-49 truthy-0 signature,
  match Python's #2571 mirror, and now gate.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013CiESAwcrZ6BcVfdkumnxa

commit-id:868e1ec0

* chore(delphi): remove vestigial np.random.seed(42); carry Clojure seeding note

The global np.random.seed(42) in cluster_dataframe was the only `random`
reference in the module and seeded an RNG nothing ever draws from:
k-means init is first-k-distinct (deterministic by construction), and
cluster_dataframe is not on the production path anyway (production uses
kmeans_sklearn exclusively; the sole caller is tests/test_clusters.py).
Also drops the module's dead `import random`.

Evidence (2026-07-05, scratch/determinism_check.py + journal): 5
consecutive full-pipeline runs on vw + biodiversity are bit-for-bit
identical except math_tick (a wall-clock version counter, varies by
design). The determinism comes from first-k-distinct + n_init=1 +
explicit random_state, not from this global seed.

Per Julien: the original Clojure author's verbatim note on seeding
(pca.clj:80-81 — "Should really throw a parallelizable random number
generator in the equation here... With seeds fed in and persisted...
XXX") now lives in pca.py next to the random_state setting, with the
seeding-history context (Clojure never fixes a seed; k-means
deterministic; PCA cold-start rand unseeded, warm-started after).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013CiESAwcrZ6BcVfdkumnxa

commit-id:7c53868d

* feat(delphi): powerit-pca port (Clojure-parity PCA) behind legacy-mode flag

Numpy port of Clojure's power-iteration PCA (pca.clj:38-105): per-component
power iteration with fixed iteration count (iters=100, the Clojure default),
Gram-Schmidt deflation, and a start_vectors parameter for warm-start pinning
(the replay harness's mechanism for collapsing cold-start jitter, and a
prerequisite for the pure-Python R2 replayer — see
docs/REPLAY_HARNESS_DESIGN.md section 9/10).

Flag: POLISMATH_PCA_IMPL env var, read at call time in
pca_project_dataframe. 'powerit' (DEFAULT - legacy/parity mode) or
'sklearn' (the improved path, previous behavior). First instance of the
permanent legacy-vs-improved dual-path pattern (Julien, 2026-07-05).
Imputation and sparsity scaling are untouched; only the eigen-solver
branches. Data-prep parity verified: Clojure imputes nil -> column average
(conversation.clj:360-380) = Python's nanmean imputation.

Documented decision - deterministic cold start: Clojure draws an UNSEEDED
random start vector (pca.clj:79-82); Python defaults to a fixed
deterministic start so the pipeline stays bit-for-bit reproducible (the
2026-07-05 determinism verification is a project invariant). Power
iteration converges to the same dominant eigenvector for almost any start,
so the fixed start is one specific draw of Clojure's random one;
start_vectors overrides for pinning. Carries the requested TODO: switch to
a proper convergence criterion once we move to improving the Python
implementation.

Silhouette guard (companion fix — required by making powerit the default):
- calculate_silhouette_sklearn (clusters.py) now treats any
  n_labels >= n_samples clustering as undefined and returns the neutral 0.0
  sentinel instead of letting sklearn raise. sklearn's silhouette_score
  requires 2 <= n_labels <= n_samples - 1; the old guard only covered
  n_labels <= 1 / n_samples <= 1.
- Why it surfaced here: making powerit the default projects some small
  conversations onto exactly two base clusters, which feeds a 2-point /
  2-label silhouette call in the group-cluster k-selection loop
  (conversation.py). That raised "Number of labels is 2. Valid values are 2
  to n_samples - 1", crashing TestConversation.test_recompute and erroring 8
  test_serialization_unfolding cases. All green now under BOTH powerit and
  sklearn.
- The new guard is a strict superset of the old one — valid clusterings
  (n_labels < n_samples) are unchanged, and with only two base clusters the
  k-selection loop is forced to k=2 regardless, so the selected clustering is
  identical; the fix only removes the crash.
- 3 unit tests added: tests/test_clusters.py::TestCalculateSilhouetteSklearn
  (2-samples/2-labels returns 0.0 not raise; single-label stays 0.0; a valid
  3-sample/2-label clustering still returns a genuine score).

Results:
- 17 new tests (tests/test_powerit_pca.py): correctness vs numpy eigh,
  deflation orthogonality, determinism, start-vector honoring, flag
  behavior, solver agreement (PC1 0 deg, PC2 2.1e-3 deg on the reference
  fixture).
- CCR angle deltas vs sklearn baseline: <= 0.02 deg across all datasets
  and variants; 28 passed / 27 xfailed unchanged.
- Evidence: the bg2050-incremental PC2 miss (10.7086 deg > 10 deg) is
  BIT-IDENTICAL under powerit - upstream incremental-state divergence,
  NOT solver difference. Its xfail annotation stands.
- Benchmark (vw, 69x125): powerit 0.765 ms avg vs sklearn 1.007 ms -
  ~1.3x faster at this size. bench_pca.py --compare-impls added.
- Final gate: 304 passed / 33 skipped / 116 xfailed / 0 failed
  (baseline 283/33/116 + 17 new + 4 benchmark-import), plus the 3
  silhouette-guard unit tests above.

Also rewords the determinism-evidence pointer to cite the journal entry
(tracked) instead of the gitignored scratch script (Copilot on #2590,
convergent with internal review).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013CiESAwcrZ6BcVfdkumnxa

commit-id:0ae4ef55

* docs(delphi): parity journal 2026-07-04/07, PLAN refresh, replay-harness design

Three docs deliverables from the 2026-07-04/05 host sessions:

- CLJ-PARITY-FIXES-JOURNAL.md: full session entries — spr-squash
  reconciliation (closed+mergedAt:null = landed; squash titles lie),
  Copilot triage of all 83 threads (1 real blocker + 5 verified
  escalations incl. the DynamoDB consensus data-loss), review-fix PR
  #2586, per-variant xfail scoping (which unmasked bg2018/pakistan
  incremental consensus divergences the blanket had absorbed), gid
  label-swap root cause + fix (#2589), seed removal (#2590),
  determinism verification (5 runs bit-identical except math_tick),
  Clojure randomness facts (never seeds; fixed-iteration power
  iteration; unseeded :twister sampling), R2 python-only constraint,
  powerit-pca GO.
- PLAN_DISCREPANCY_FIXES.md: D10/D11/D12 status rows corrected (were
  still "VM draft — NEEDS REWORK"; actually code-complete open PRs).
- REPLAY_HARNESS_DESIGN.md (NEW): design for the replay harness (H) —
  schedule spec as first-class input, Python driver (the future R2
  forward model, python-only per Julien 2026-07-05), Clojure driver
  narrowed to R1 certification (Mode A pure conv-update reduce; blob
  capture sufficient, EDN on demand), nondeterminism policy
  (tolerance classes, warm-start pinning, self-jitter measurement),
  storage/provenance, phased build plan H-0..H-D.

Also appends the 2026-07-06/07 session entry: the silhouette-guard fix
for the powerit-PCA default (#2591) — powerit collapsing a small
conversation to two base clusters fed a 2-point/2-label silhouette call
that sklearn rejects; calculate_silhouette_sklearn now returns 0.0 when
n_labels >= n_samples (strict superset of the old guard, chosen
clustering unchanged). Verified green; details in the journal.

The journal's "Determinism verification" entry is the tracked evidence
pointer cited from pca.py (Copilot on #2590).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013CiESAwcrZ6BcVfdkumnxa

commit-id:d2fa477e

* feat(delphi): Clojure-parity math fixes + storage-v2/replay design (squash of 15 PRs)

SQUASH MERGE of the 15-PR Clojure-parity + design stack into edge
(spr stack, PRs #2564–#2597). GitHub's default squash message keeps only
the top PR's description; this message preserves every squashed PR's commit
message so no history is lost.

What it does:
  • Clojure-parity math fixes D10–D15 and supporting changes (rep-comment
    selection, consensus selection, comment priorities, group-id order,
    ns/PASS-vote counting, repness cleanup) so the Python math pipeline
    matches the legacy Clojure output.
  • powerit-pca legacy-mode PCA port (Clojure-parity power iteration).
  • DynamoDB consensus serialization/robustness fixes (Decimal conversion,
    dict-shape normalization incl. present-but-None guard).
  • Test-infra: require_dynamodb skips locally / fails loudly in CI.
  • Docs: parity journal + PLAN refresh, replay-harness design (R1/R2),
    storage-v2 / job-id reproducibility design, and archival of stale docs.

───────────────────────────────────────────────────────────────────────
Squashed PRs (bottom → top of stack):
───────────────────────────────────────────────────────────────────────

▁▁▁ #2564 ▁▁▁
refactor(delphi): delete dead scalar paths in repness.py (PR 14a)

The scalar implementations in `delphi/polismath/pca_kmeans_rep/repness.py`
were test-only — production calls only `compute_group_comment_stats_df`,
`select_rep_comments_df`, and `select_consensus_comments_df` (via
`conv_repness`). Maintaining two parallel implementations created
"where do I put this helper?" ambiguity for the upcoming D10/D11/D12 fixes
(all of which add new helpers to `repness.py`) and obscured the structure
of the production path.

Foundation pass before D10/D11/D12 + PR 14b/14c. No behavioural change —
production path untouched.

## Production code (`delphi/polismath/pca_kmeans_rep/repness.py`)

DELETED (445 lines):
- Primitives: `prop_test`, `two_prop_test` (no production callers).
- Orchestration: `comment_stats`, `add_comparative_stats`, `repness_metric`,
  `finalize_cmt_stats`, `passes_by_test`, `best_agree`, `best_disagree`,
  `select_rep_comments`, `select_consensus_comments`.
- Unused helper: `calculate_kl_divergence` (no callers anywhere in repo).

KEPT:
- `z_score_sig_90`, `z_score_sig_95` — trivial threshold checks consumed
  scalar-side; vectorizing would not save lines.

ENRICHED:
- `prop_test_vectorized` and `two_prop_test_vectorized` docstrings now embed
  the scalar-equivalent closed-form algebra. The formulas stay readable even
  though the scalar functions are gone.

## Tests

DELETED entirely:
- `tests/test_old_format_repness.py` (557 lines, scalar-only sibling of
  `test_repness_unit.py`).

DELETED classes/methods in `tests/test_repness_unit.py`:
- `TestCommentStats`, `TestSelectionFunctions`, `TestConsensusAndGroupRepness`
  (all scalar-only).
- `TestStatisticalFunctions::test_prop_test`, `::test_two_prop_test`.

MIGRATED to single vectorized DataFrame calls in `tests/test_discrepancy_fixes.py`:
- D4/D5/D6 BlobInjection classes — build a DataFrame from the blob's `repness`
  entries, run one `prop_test_vectorized` / `two_prop_test_vectorized` call,
  compare element-wise. Tests the actual production code path, produces
  cleaner diagnostics via `.to_string()`.
- `TestD5ProportionTest::test_prop_test_matches_clojure_formula` (consolidated
  with the n=0 boundary case).
- `TestD6TwoPropTest::test_two_prop_test_matches_clojure_formula` (+ edge cases
  consolidated, + pi_hat=1 boundary cases).

CONSOLIDATED in `tests/test_discrepancy_fixes.py`:
- `TestD8FinalizeStats`'s 7 scalar boundary tests collapse into a single
  parametrized DataFrame test `test_repful_classification_boundary` that
  exercises the production `np.where(rat > rdt, 'agree', 'disagree')` logic.
  All boundary cases preserved (rat<rdt, rat>rdt, rat==rdt non-zero,
  rat==rdt==0, negative z-scores).

DELETED redundant tests:
- `TestSyntheticEdgeCases::test_prop_test_matches_clojure_formula_synthetic`
  (duplicated by migrated TestD5ProportionTest).
- `TestSyntheticEdgeCases::test_clojure_repness_metric_product` (duplicated
  by migrated TestD7RepnessMetric).
- `TestSyntheticEdgeCases::test_clojure_repful_uses_rat_vs_rdt` (purely
  tautological).

Cross-checks in `tests/test_repness_unit.py::TestVectorizedFunctions` now
use `_prop_test_reference` and `_two_prop_test_reference` closed-form
staticmethods in place of scalar calls.

## Suite delta

- Pre (edge @ 2dce7385f): 330 passed, 12 skipped, 58 xfailed.
- Post:                    295 passed, 12 skipped, 58 xfailed.
- Delta: -35 passed, 0 failed, 0 new xfailed. Matches deleted scalar-test
  count exactly.

## For PR 14c (readability refactor — runs later)

The deleted scalar code is the readability reference for PR 14c. Retrieve via:

    git show <this-commit>~1:delphi/polismath/pca_kmeans_rep/repness.py \
        | sed -n '161,302p'

Specifically (pre-deletion line numbers): `comment_stats` 161-201,
`add_comparative_stats` 203-235, `repness_metric` 237-271,
`finalize_cmt_stats` 273-301. Clojure originals at
`math/src/polismath/math/repness.clj:78-100,173-188,191-200`.

## Documentation

- `delphi/docs/PLAN_DISCREPANCY_FIXES.md`: added PR 14a row to the stack
  cross-reference table.
- `delphi/docs/CLJ-PARITY-FIXES-JOURNAL.md`: appended "Session: PR 14a —
  Scalar deletion (2026-06-11)" entry with the full scope, suite delta,
  and the `git show` recipe for PR 14c.

## Out of scope (handed off separately)

- Pyright pandas-stubs noise (10+ false positives on `pd.DataFrame(columns=...)`
  and `df['col'] = value` in `compute_group_comment_stats_df`) is pre-existing
  on edge HEAD; PR #2560's pyright config didn't set rule overrides.
  Handoff: `~/polis/HANDOFF_PYRIGHT_PANDAS_STUBS.md`. Tracked separately.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

▁▁▁ #2570 ▁▁▁
fix(delphi): ns includes PASS votes (Clojure parity, pre-D10)

Bug
---
`compute_group_comment_stats_df` in
`delphi/polismath/pca_kmeans_rep/repness.py` computed `ns = na + nd` and
`total_votes = total_agree + total_disagree`, silently dropping PASS (0)
votes from every vote count.

Clojure reference: `math/src/polismath/math/repness.clj:56-61, :70`:

    (defn- count-votes [votes & [vote]]
      (let [filt-fn (if vote #(= vote %) identity)]
        (count (filter filt-fn votes))))
    ...
    :ns (fnk [votes] (count-votes votes))

`count-votes` with no `vote` argument uses `identity` as the filter
predicate. In Clojure 0 is truthy, so `(filter identity ...)` keeps every
non-nil entry — including PASS. Therefore Clojure
`ns = na + nd + np` (PASS count).

Every downstream metric that consumes `ns` — `pa`, `pd`, `pat`, `pdt`,
`ra`, `rd`, `rat`, `rdt`, `agree_metric`, `disagree_metric`, and the
upcoming D11 `consensus_stats_df` — was off whenever PASS votes existed.

Fix
---
Both `total_counts` and `group_counts` now compute the count column via
`('vote', 'size')` directly in the pandas groupby aggregation. The frame
is `dropna(subset=['vote'])`-filtered upstream, so `size()` counts
exactly the non-NaN rows — including PASS (0). This is the Clojure
`(count (filter identity votes))` recipe verbatim. Both sites carry a
comment citing the Clojure reference.

Why D5 BlobInjection didn't catch this
--------------------------------------
D5's blob-injection tests pull `(n-success, n-trials)` directly from the
Clojure blob's `repness` entries and feed them to `prop_test_vectorized`.
They bypass `compute_group_comment_stats_df` entirely, so anything
downstream of `ns = na + nd` was invisible. The same gap will recur for
every fix whose stage-inputs are built by Python code. Pure-formula
tests over a tiny vote matrix are the only RED path.

Tests added
-----------
`TestNsIncludesPassVotes` in `tests/test_repness_unit.py`:

- `test_ns_includes_pass_votes` — single comment, 2 agree / 1 disagree /
  2 pass → `na=2, nd=1, ns=5`.
- `test_ns_all_pass_column` — all-PASS column → `na=0, nd=0, ns=3`.
- `test_ns_mixed_with_nan_only_explicit_votes_count` — NaN never counts;
  PASS always does.
- `test_other_votes_includes_other_group_pass` — `other_votes` (the
  complement of group `ns`) also includes out-of-group PASS.

Suite delta
-----------
Pre: 295 passed, 12 skipped, 58 xfailed (PR 14a baseline).
Post: 299 passed, 12 skipped, 58 xfailed. Delta = +4 (the new tests).
No pre-existing test broke — the existing `TestVectorizedFunctions`
fixtures used only AGREE/DISAGREE/NaN and were never sensitive.

Cascade to D11
--------------
D11's `consensus_stats_df(vote_matrix_df)` (whole-conversation
counterpart) will mirror the same recipe at the conversation level. This
fix lands BEFORE D10 in the stack so D11's implementation, when it
arrives, starts from the corrected ns semantics. D11 should follow the
same pure-formula test pattern.

Goldens
-------
DEFERRED. Re-recording is gated on sklearn-KMeans-seeding consensus; no
golden values shift at the goldens commit until D10/D11/D12 land.

▁▁▁ #2566 ▁▁▁
feat(delphi): D10 — Clojure-parity rep comment selection (PR 8)

Replaces the pre-D10 botched-port `select_rep_comments_df` with a
single-pass reduce that mirrors Clojure `select-rep-comments`
(math/src/polismath/math/repness.clj:212-281).

Sits on top of PR 14a in the spr stack.

## Helpers added (top-level in `repness.py`)

- `passes_by_test(s)` — Clojure `passes-by-test?` (repness.clj:165-170).
  OR'd on (rat, pat) and (rdt, pdt) z-sig-90. NO `pa >= 0.5` gate; the
  pre-D10 Python gate was an over-restriction with no Clojure analog.

- `beats_best_by_test(s, current_best_z)` — Clojure `beats-best-by-test?`
  (repness.clj:133-139). Strict `>` on `max(rat, rdt)`.

- `beats_best_agr(s, current_best)` — Clojure `beats-best-agr?`
  (repness.clj:142-162). Four-branch agree-priority logic:
    1. na == 0 AND nd == 0 → reject.
    2. current_best AND current_best.ra > 1.0 → compare 4-way signed product
       `ra * rat * pa * pat`.
    3. current_best (else, ra <= 1.0) → compare `pa * pat` only.
    4. No current_best → accept if `z90(pat)` OR `(ra > 1.0 AND pa > 0.5)`.
  `current_best` stores the RAW row (Clojure repness.clj:250) so the
  ra/rat/pa/pat surface stays available across iterations.

- `_finalize_row_for_output(row, *, is_best_agree=False)` — Clojure
  `finalize-cmt-stats` (repness.clj:173-188) + best-agree flagging
  (repness.clj:262-264). Emits `best_agree=True` and `n_agree=na` for the
  best-agree slot.

## `select_rep_comments_df` rewrite

Signature now: `(stats_df, mod_out=None) -> List[Dict[str, Any]]`. Drops
the `agree_count` / `disagree_count` kwargs (Clojure has only a cap of 5).

Per-row state `{sufficient, best, best_agree}` updated by the helpers.
Final assembly: dedup best_agree from sufficient → sort by metric
(agree_metric for repful=='agree', disagree_metric for 'disagree')
→ prepend finalized+flagged best_agree → take 5 → agrees-before-disagrees.

The caller in `conv_repness` drops the `_stats_row_to_dict` wrapping step
(the new function returns finalized dicts directly).

## Two pre-D10 bugs fixed alongside the rewrite

- `pa >= 0.5 / pd >= 0.5` over-gate in the passing filter — removed (no
  Clojure analog).
- "Fill from other category" + "first row" fallback blocks — deleted. The
  `:best` / `:best_agree` mechanism IS the Clojure fallback.

## Tests (18 new in `tests/test_discrepancy_fixes.py`)

- TestD10PassesByTest (4): agree-side, disagree-side, neither, no pa-gate.
- TestD10BeatsBestByTest (3): None-best, max(rat,rdt), strict `>`.
- TestD10BeatsBestAgr (6): one per Clojure branch + boundary.
- TestD10SelectRepCommentsBoundary (5): empty input, single unvoted row
  → best fallback, sufficient-empty-best-agree-only, take-5 cap +
  agrees-before-disagrees, **the eviction edge case** (best_agree outside
  sufficient evicting 5th-highest-metric).

## Eviction edge case — flagged

`take(5)` runs AFTER prepending best_agree. If `best_agree` was kept by
`beats_best_agr` as a non-significant agree-priority fallback (failed
`passes_by_test`, qualified via Branch 4) AND `:sufficient` already has 5
entries, the prepend pushes total to 6 and `take(5)` evicts the
5th-highest-metric sufficient entry — possibly a strong dissenting view.

Mirrors Clojure exactly for blob parity. `# TODO(parity-eviction)` comment
at the take(5) site in the production code; entry added under
"Pending — needs team discussion" in PLAN.md; pinned by the synthetic
test above.

## Re-xfailed with updated reasons (D14 / D1 upstream divergence)

Six per-shared-(gid, tid) blob-comparison tests were previously xfailed as
"D5/D6/D7/D8/D10: no shared comments to compare". After D10 there ARE
shared comments (overlap ~20% on vw cold_start), but the per-(gid, tid)
stats still mismatch because Python and Clojure place different
participants in the "same" group ID. That's upstream PCA/KMeans
group-membership divergence (D14 / D1), not D10. Updated xfail reasons
point at D14 / D1. D10 itself is verified by the 18 synthetic tests.

Affected: TestD9ZScoreThresholds::test_z_values_match_clojure,
TestD5ProportionTest::test_pat_values_match_clojure_blob,
TestD6TwoPropTest::test_rat_values_match_clojure_blob,
TestD7RepnessMetric::test_repness_metric_matches_clojure_blob,
TestD8FinalizeStats::test_repful_matches_clojure_blob,
TestD10RepCommentSelection::test_rep_comments_match_clojure.

## Suite delta

- Pre (post-14a baseline): 295 passed, 12 skipped, 58 xfailed.
- Post:                    313 passed, 12 skipped, 58 xfailed.
- Delta: +18 passed (the 18 new D10 synthetic tests), 0 failed, no new
  xfailed.

## Documentation

- `delphi/docs/PLAN_DISCREPANCY_FIXES.md`: PR 14a row marked landed
  (#2564), PR 8 (D10) marked in-flight, eviction concern added to
  "Pending — needs team discussion".
- `delphi/docs/CLJ-PARITY-FIXES-JOURNAL.md`: "Session: PR 8 — D10 rep
  comment selection (2026-06-11)" entry with full scope, suite delta,
  and decisions log pointer.

## /goal mode

This PR is part of an autonomous run (`/goal`) targeting D10 + D11 + D12 +
golden snapshots as a stacked PR series. Decisions made autonomously
(key naming, return type, etc.) are documented in
`~/polis/D10_D11_D12_GOLDENS_DECISIONS.md` for batch user review at the
end of the run.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

▁▁▁ #2567 ▁▁▁
feat(delphi): D11 — Clojure-parity consensus comment selection (PR 9)

Replaces the pre-D11 consensus logic (per-group `pa > 0.6 for all` filter,
top 2 by `avg_pa`) with a whole-conversation per-comment-stats stage and
two independent top-5 lists (agree / disagree), matching Clojure
`consensus-stats` + `select-consensus-comments`
(math/src/polismath/math/repness.clj:284-323).

Sits on top of PR 8 (D10) in the spr stack.

## Production changes (`repness.py`)

- **`consensus_stats_df(vote_matrix_df, mod_out=None) -> pd.DataFrame`**:
  new helper. Whole-conversation per-comment stats (no group split). Output
  DataFrame indexed by tid with cols [na, nd, ns, pa, pd, pat, pdt].
  Vectorized port of Clojure `consensus-stats` (repness.clj:284-290).

  **`ns` includes PASS** (Clojure parity, repness.clj:56-61): computed as
  `vote_matrix_df.notna().sum(axis=0)` rather than `na + nd`. Matches
  Clojure `(count (filter identity ...))` where `0` (PASS) is truthy.

- **`select_consensus_comments_df` rewrite**: new signature
  `(cons_stats) -> Dict[str, List[Dict]]`. Filters and ordering:
    - agree: `pa > 0.5 AND z-sig-90(pat)`, sorted desc by `am = pa * pat`.
    - disagree: `pd > 0.5 AND z-sig-90(pdt)`, sorted desc by `dm = pd * pdt`.
  Cap: top 5 each side. Output: `{'agree': [...], 'disagree': [...]}`.
  With PSEUDO_COUNT=2, `pa + pd = 1` exact → `pa > 0.5 ⟺ pd < 0.5`, so
  the same tid cannot appear in both lists.

- **`conv_repness` grows `mod_out` kwarg**, forwarded to both
  `select_rep_comments_df` and `consensus_stats_df`. Consensus is now run
  unconditionally — Clojure has no `len(groups) > 1` guard.

- **`_stats_row_to_dict` deleted** — orphan after D11.

## Caller (`conversation.py`)

`_compute_repness` passes `mod_out=self.mod_out_tids` to `conv_repness`,
matching Clojure's mod-out propagation (repness.clj:222 and :296).

## Downstream output shape change

`conv.repness['consensus_comments']` was a flat list with
`{repful: 'consensus', comment_id, avg_agree, stats}` entries. After D11
it's `{'agree': [entries], 'disagree': [entries]}` matching Clojure's
math-blob shape. Each entry has Python-convention keys (decision S1):
`{comment_id, n_success, n_trials, p_success, p_test}`.

Updated consumers in the test suite:
- `tests/test_repness_smoke.py::test_repness_structure` — iterates the
  new dict shape.
- `tests/test_pipeline_integrity.py::test_full_pipeline` — same.

External downstream consumers (`client-report/normalizeConsensus.js`) may
need a parallel update; flagged in the decisions log for batch review.

## Tests (13 new in `tests/test_discrepancy_fixes.py`)

- `TestD11ConsensusStatsDf` (5): basic counts, pseudocount pa/pd
  smoothing, ns=0 uninformative fallback, mod_out tid filter,
  **ns-includes-PASS Clojure parity**.
- `TestD11SelectConsensusBoundary` (8): empty input, clear agree
  consensus, clear disagree consensus, divisive (no consensus), top-5
  cap, entry-key shape, disagree-side key mapping (n_success ← nd,
  p_success ← pd, p_test ← pdt), mutually-exclusive agree/disagree lists.

## `ns`-PASS fix (Clojure parity)

After D11 was first landed (PR #2567), the real-data test
`test_consensus_matches_clojure` showed 3-5/5 overlap on cold_start.
Investigation revealed that Clojure's `:ns` (via `count-votes` with
`filter identity` — repness.clj:56-61) INCLUDES PASS votes (`0` is truthy
in Clojure), while Python's `ns = na + nd` excluded them.

Fixed here by switching `consensus_stats_df` to count via
`vote_matrix_df.notna().sum(axis=0)`. After the fix, 3 of 4 dataset
variants (vw-incremental, vw-cold_start, biodiversity-cold_start) match
Clojure exactly. The `biodiversity-incremental` variant still mismatches
on the disagree side (likely residual upstream PCA/KMeans group-membership
divergence) — remains `xfail(strict=False)` and tracked in the journal.

(The companion fix for `compute_group_comment_stats_df` ships in a
separate pre-D10 commit `qyskkqkovtmn`.)

## B1 + B2 sub-agent fixes (relocated from D12 per batch review 2026-06-11)

These two fixes were originally landed in PR #2568 (D12) because the D11
sub-agent review happened AFTER D11 had been pushed. They belong in D11,
so they are squashed into this commit:

- **B1** (`polismath/conversation/conversation.py:830-837`): the
  no-groups branch of `_compute_repness` returned
  `'consensus_comments': []` (list). After D11, the public shape is the
  dict `{'agree': [...], 'disagree': [...]}`. Downstream consumers
  (test_repness_smoke, test_pipeline_integrity) iterate the dict shape
  and would crash on the legacy list. Fixed to always return the dict
  shape, even with no groups.

- **B2** (`tests/test_legacy_repness_comparison.py:197-205`): the
  legacy-comparison test extracted `py_consensus = py_results.get(
  'consensus_comments', [])` and treated it as a flat list. Post-D11
  this is a dict, so the ID extraction silently produced an empty set.
  Fixed to flatten the dict (agree + disagree) for the legacy comparison,
  with a `legacy fallback` branch in case the value is still a list.

## Suite delta

- Pre (post-D10):    313 passed, 12 skipped, 58 xfailed.
- Post (this PR):    330 passed, 12 skipped, 55 xfailed, 3 xpassed.
- Delta: +17 passed, -3 xfailed (D11 real-data test now passes on 3 of
  4 dataset variants; biodiversity-incremental remains
  xfail(strict=False)). Zero regressions.

## /goal mode

Part of an autonomous stacked PR series (D10 + D11 + D12 + goldens) per
user request. Decisions are documented for batch review in
`~/polis/D10_D11_D12_GOLDENS_DECISIONS.md`.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

▁▁▁ #2568 ▁▁▁
feat(delphi): D12 — comment priorities (PR 11)

Implements `comment-priorities` matching Clojure
(math/src/polismath/math/conversation.clj:648-679). Pre-D12 Python emitted
nothing for `comment_priorities`, which forced the TypeScript server's
`getNextPrioritizedComment` to fall back to uniform random comment routing.

Sits on top of PR 9 (D11) in the spr stack.

## New helpers in `pca.py`

- `pca_project_cmnts(center, comps) -> np.ndarray` (shape (n_cmnts, n_components)):
  Vectorized projection. Closed-form derived from Clojure's sparsity-aware
  projection (pca.clj:134-178) collapsing to a single non-nil column per
  comment:
      proj[i] = -sqrt(n_cmnts) * (1 + center[i]) * [pc1[i], pc2[i]]

- `compute_comment_extremity(cmnt_proj) -> np.ndarray`: L2 norm per row.
  Clojure `with-proj-and-extremtiy` (conversation.clj:341-352).

## New module-level functions in `conversation.py`

- `META_PRIORITY = 7` (Clojure conversation.clj:319).
- `importance_metric(A, P, S, E) -> float` (Clojure conversation.clj:311-315).
- `priority_metric(is_meta, A, P, S, E) -> float` (Clojure conversation.clj:321-330).
  Squared output. Meta: `META_PRIORITY^2 = 49`. Non-meta:
  `(importance * (1 + 8*2^(-S/5)))^2` — the decay factor lets new comments
  bubble up; importance falls as votes accumulate.

## New `Conversation._compute_comment_priorities()` method

Wired into `recompute()` after `_compute_repness()`. For each tid:
- Compute extremity from `pca_project_cmnts` + `compute_comment_extremity`.
- Aggregate A/D/S across all groups via `_compute_group_votes()`.
- Derive P = S - (A + D)  (Clojure conversation.clj:661).
- Check `tid in self.meta_tids` for the meta branch.
- Call `priority_metric` and store under `self.comment_priorities[int(tid)]`.

The serialization infrastructure (`to_dict`, `to_dynamo_dict`,
underscore→hyphen conversion in `_convert_inner`) already existed but was
emitting empty. Now populated.

## B1 + B2 fixes folded in from D11 sub-agent review

- **B1**: `conversation.py:834` no-groups early-return now emits
  `consensus_comments: {'agree': [], 'disagree': []}` (dict) instead of
  `[]` (list) — restores shape consistency with the new D11 shape.
- **B2**: `test_legacy_repness_comparison.py:197` flattens the new
  consensus dict before iterating, instead of crashing on
  `'agree'.get('comment_id', '')`.

## Tests (11 new in `tests/test_discrepancy_fixes.py`)

- `TestD12PCAProjectComments` (5): output shape, formula verification per
  row, empty inputs, L2 extremity, empty extremity.
- `TestD12PriorityMetrics` (6): `importance_metric` matches Clojure
  reference value `4/9` (from conversation.clj:335 comment), extremity
  boost behavior, meta priority constant = 49, non-meta squared formula,
  decay factor monotonicity (low-S boosts, high-S fades), `META_PRIORITY == 7`.

## Real-data test xfailed: Clojure blob has constant priorities

vw and biodiversity blobs both have EVERY tid set to priority = 49.0
(= META_PRIORITY^2). Likely caused by Clojure's `(if 0 ...)` truthiness
quirk — 0 is truthy in Clojure (only nil/false are falsy), so any value
returned by `(get meta-tids tid 0)` triggers the meta branch.

Python correctly distinguishes meta from non-meta via Boolean set
membership, producing varied priorities 0.18-31.46. Spearman comparison
meaningless when Clojure side has zero variance (returns nan).

Test xfailed with full reason. D12 logic verified by the 11 synthetic
tests. Logged for batch review — Python may be MORE correct than Clojure
here.

## Suite delta

- Pre (post-D11): 325 passed, 12 skipped, 58 xfailed.
- Post (this PR): 336 passed, 12 skipped, 56 xfailed, 2 xpassed.
- Delta: +11 (the 11 new D12 synthetic tests), 0 failed, 2 xfailed →
  xpassed (the cold_start D12 tests now run cleanly; the new xfail is
  on a different test).

## /goal mode

Autonomous stack PR (D10 + D11 + D12 + goldens). Decisions documented in
`~/polis/D10_D11_D12_GOLDENS_DECISIONS.md` for batch user review.

D11 sub-agent flagged addi…

…2677)

The agent is already INSTALLED on every instance (launchTemplates.ts, inside the
shared `usrdata()`), but the config download and `systemctl start` live only in
the ollama user-data block. So only the GPU box publishes `mem_used_percent` —
every other tier reports CPU, network and EBS but no memory.

That matters because memory, not CPU, is the binding resource on these
instances. A 30-day production baseline shows the math worker at 0.5% mean CPU
and both delphi tiers at 0.6-1.3%, so CPU says only "idle"; without memory there
is no evidence on which to right-size them.

This moves the agent's config-and-start block from `ollamaUsrData` into the
shared `usrdata()`. Effect: every instance publishes `mem_used_percent` and
`disk used_percent`. The `nvidia_gpu` section of the config collects nothing
where there is no GPU, so the ollama box is unaffected.

Safety notes, both deliberate:

* All five launch templates already use the same `instanceRole`, and
  `cwAgentConfigAsset.grantRead(instanceRole)` is already granted — so the S3
  fetch is authorised everywhere. Verified, not assumed.
* `usrdata()` runs under `set -e`. The added commands are therefore guarded with
  `|| true` and `|| echo`: a metrics agent must never be able to abort an
  instance boot. This is the one place this deliberately differs from the ollama
  block it is derived from, which is unguarded.

Not included on purpose: no change to the agent config JSON, no new metrics, no
change to collection interval. This turns on what is already configured.

Verified before submitting:

* `npx tsc --noEmit` exits 0, before and after. `usrdata` is declared above
  `cwAgentConfigAsset` but only CALLED below it, and TypeScript does not object.
* `npx cdk synth` succeeds with no AWS credentials.
* In the synthesized CloudFormation, all five launch templates contain
  `systemctl start amazon-cloudwatch-agent` exactly once, each guarded. Ollama
  gets it once, not twice — the move is not a duplication.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

…d py cache + parallel entries (Phase 0)

## Why

`GOAL_CUTOVER_READY.md` Phase 0 (Julien ruling 2026-07-27): the certification battery re-ran the full ~36-minute Python re-replay after ANY `polismath` edit, blocking every code change.

## What

- The Python recording cache key is now the ENGINE-scoped tree hash: pure-harness files (`certify.py`, `poller_equiv.py`, `prodclone.py`, `shard_bench.py`, `poller/**`) are excluded from the hash — none is reachable from the replay subprocess import graph (`scripts/replay_driver.py` -> driver/schedule/real_data/store/stepcompare/types -> engine). Manifest key renamed `py_tree_sha256` -> `engine_tree_sha256`; the resulting one-time invalidation of all 20 Python recordings doubled as the timed A/B run.
- `run_battery(workers=N)` fans the per-entry heavy work (driver subprocesses + hash-first compare) across threads; the ledger fold stays strictly serial in battery order, so report + ledger are bit-identical to `workers=1` (pinned by test). Step-verdict cache writes are atomic (tmp + `os.replace`). CLI `--workers` defaults to 6.

## Measured A/B on the 10-core host

- First pass: 19m17s wall vs ~36m serial (the long pole: `pakistan:uniform8` alone is ~18m).
- Cached pass: 22s.
- Harness-only edit: 2m9s with ZERO re-replays (previously: the full 36-minute re-replay).

## Testing

TDD: 7 RED -> GREEN; replay_harness 464 passed / 5 skipped (baseline 456/5 plus exactly the 8 new tests). Battery on this tree: 20/20 MATCH x2 (post-invalidation re-record + cached pass). The Phase 1 `engine_mode` inventory (18 branch sites classified DELETE/PARK/KEEP) is journaled.

commit-id:31c19ea5